Does Human Action Recognition Benefit from Pose Estimation?

Introduction The earliest works in action recognition focused on tracking body parts and classifying the joint movements. These pose-based approaches, while straight-forward, require accurate tracking of body parts, which is a challenging task in its own right. As recent trends in action recognition have shifted towards natural and unconstrained videos (e.g. films, broadcast sports, Youtube videos), efforts have shifted from high-level modelling of the human body to directly classifying actions with abstract and low-level appearance features in appearance-based approaches. But despite requiring more initial processing, pose representations have several advantages. First, they have fewer intra-class variances; in particular, 3D skeleton poses are viewpoint and appearance invariant, such that actions vary less from actor to actor. Secondly, using pose representations simplifies learning for action recognition, since relevant highlevel information has already been extracted. Given the great progress in pose estimation over the past few years [1], we feel that pose-based action recognition systems warrant a second look. In this work, we compare pose-based and appearance-based features for action recognition as depicted in Fig. 1. Our pose-based features are derived from articulated 3D joint information; we label as appearancebased any feature which can be extracted from video data without explicit articulated modelling of the human body. For fair comparison, we apply the same action recognition system [4] to the two different sets of features. Finally, we combine the two feature types into a single system.